Speech/Gesture Interface to a Visual-Computing Environment

Although recent years have witnessed tremendous progress in 3D immersive display and virtual reality, the corresponding humancomputer interaction (HCI) technologies have lagged behind. Current interfaces involve heavy headsets, datagloves, tethers, and other virtual reality devices, which may deter or distract the user. To fully exploit VR’s potential for visualizing and interacting with complex information, users must be able to interact with the virtual display in a more natural way (see the “Hand gestures” sidebar on the next page). In this article, we describe a bimodal speech/gesture interface, which we have integrated in a 3D visualcomputing environment used by structural biologists. This interface lets researchers interact with 3D graphical objects in a virtual environment using spoken words and simple, free hand gestures. The reason we used a particular virtual environment context was to set the necessary constraints to make our analysis robust and to develop a command language that optimally combines speech and gesture inputs. Our interface uses